Hypersfm I. . Introduction

We propose a novel algorithm that solves the Structure from Motion problem in a divide and conquer manner by exploiting its bipartite graph structure. Recursive partitioning has a rich history, stemming from sparse linear algebra and finite element methods, and are also appealing for solving large-scale SfM problems. However, an important and less explored question is how to generate good partitionings for SfM that divide the problem into fullyconstrained sub-problems. Here we introduce HyperSfM, a principled way to recursively divide an SfM problem using a hypergraph representation, in which finding edge separators yields the desired “nested-dissection” style tree of nonlinear sub-problems. After partitioning, a bottomup computation pass solves the SfM problem robustly (by having fully constrained sub-problems) and efficiently (because most nonlinear error is removed at lower levels of the tree). The performance of the algorithm is demonstrated for various indoor and outdoor standard data-sets. I. . Introduction Large-scale structure from motion (SfM) problems have gained more and more attention lately, as SfM is becoming one of the key technologies in applications such as cityscale 3D reconstruction. A lot of effort has been made to push SfM algorithms towards collections of a large number of photos [2], [3]. In this paper we concentrate on the back-end optimization phase, after feature extraction and data association has been performed, which are daunting problems in their own right [1], [4]. In photogrammetry, divide and conquer approaches are a common and popular way to “bundle” data from a large area [5]. When images are taken sequentially from a plane or a ground vehicle, a feasible way to tackle the problem is to solve the subproblem within a relative coordinate system rather than a consistent global coordinate system [6]. However, this approach is decidedly suboptimal when there are a lot of “loop closures” in the camera trajectory, which is typically the case in unstructured photo-collections. Moreover, for such unordered, wide-baseline data-sets we typically do not have knowledge of the capture ordering, making this approach unsuitable. Another important problem worth investigating is how to avoid degeneracies when generating submaps. General partitioning algorithms [7] do not take into account the domain knowledge of SfM problems. Hence, directly applying those algorithms will easily introduce degeneracies to the state variables, especially 3D points, as each 3D point is typically only visible in a small number of cameras (two or three in practice). In fact, little work has been done on how to optimally divide the SfM problem while keeping all the individual sub-problems fully constrained. In this paper, we propose a principled way to partition the SfM problem. We exploit the bipartite structure of the SfM visibility graph and convert it to a simplified camera hypergraph. It is shown that vertex separators composed of only 3D points can be located from the hypergraph, and non-singularity can be strictly enforced by imposing a graph refinement step after partitioning. Our algorithm is not only out-of-core but also naturally alleviates the initialization issue of bundle adjustment in SfM problems [8], [9]. We employ a bottom-up optimization using the submap tree obtained by recursive partitioning. The optimization over different subtrees in the same level can be carried out in parallel. The optimized submaps are aligned to one another after passing information up the tree to their common ancestor. As a beneficial consequence we never need to generate the initialization for the entire large-scale problem. In the results section, we demonstrate the effectiveness of our algorithm using several indoor and outdoor data sets from Microsoft’s PhotoSynth database. We show that the proposed partitioning scheme effectively decouples original problems, and that the resulting sub-problem structure greatly speeds up the bottom-up bundle adjustment phase. II. . Background and Related Work